Task execution system, task execution method, training apparatus, and training method

ABSTRACT

A system that uses a learning module to realize execution of a predetermined task includes: a first input unit configured to receive information acquired from one or more external systems, and generate at least a portion of information to be input to the learning module; an output unit configured to acquire information output from the learning module, and generate information to be output from the system, the information output from the system being information based on which execution of a predetermined task is to be realized; and a second input unit configured to receive an input from a user so that information based on the input from the user is input to at least one of the first input unit, the learning module, and the output unit, and information output from the output unit varies based on the input from the user.

CROSS-REFERENCES TO RELATED APPLICATION

This application claims priority to Japanese Patent Application No.2017-093222 filed May 9, 2017, the entire contents of which areincorporated herein by reference.

FIELD

The disclosure relates to a system that realizes execution of a task, amethod for realizing execution of a task, and a training apparatus and atraining method for the same.

BACKGROUND

Conventionally, it is known that machine learning technology including aneural network is used to control a system to cause the system toexecute a predetermined task. For example, JP 2017-20135A discloses thatmachine learning is applied to the picking of grip-target objects thatare piled up in bulk, so that target objects with a high grippingsuccess rate are learnt, and picking is performed. Also, for example, JP3978098B2 discloses that machine learning (with a rule-based classifier)is applied to classification processing that is performed to determinewhether or not samples have a defect, using captured images, and a usersets the configuration of the classifier, in advance of learning.

JP 2017-20135A and JP 3978098B2 are examples of background art.

SUMMARY

At a work site, there are cases where conditions such as requirementsand constraint conditions concerning work, which are unique to everysite, are adjusted during the execution of work, according to accuracy,execution speed, failure tolerance, and so on that are required whenwork is executed. However, with a system that uses a trained model torealize execution of a predetermined task (hereinafter also referred toas “work”), operations of the system are determined based on inputs fromdevices such as sensors so that work is performed. Therefore, in orderto make adjustments according to the conditions of work, it is necessaryto re-train the trained model, and it is impossible to adjust theconditions during the execution of work.

Therefore, one or more aspects aim to provide technology for allowing auser to make adjustments according to the conditions of work, during theexecution of work, in cases where a system realizes execution of apredetermined task using a learning module that includes a trained modelor a model that is equivalent to the trained model.

Provided is a system for executing a predetermined task, the systemincluding; a learning module including a trained model that has beensubjected to predetermined training through machine learning or a modelthat is equivalent to the trained model in terms of an input-outputrelationship; a first input unit configured to receive information thatis acquired from one or more external systems, and generate at least aportion of information that is to be input to the learning module; anoutput unit configured to acquire information that is output from thelearning module, and generate information that is to be output from thesystem, the information output from the system being information basedon which execution of a predetermined task is to be realized; and asecond input unit configured to receive an input from a user so thatinformation that is based on the input from the user is input to atleast one of the first input unit, the learning module, and the outputunit, and information that is output from the output unit varies basedon the input from the user.

According to this aspect, when the system is caused to execute apredetermined task, using a learning module that includes a trainedmodel or a model that is equivalent to the trained model, informationthat is to be output varies based on not only information acquired froman external system such as a sensor, but also information that is inputby a user. Thus, by inputting a condition for a task during theexecution of work, the user can acquire an output that has been adjustedaccording to the condition, without re-training the learning module. Asa result, the user can realize high-speed execution of a task thatsatisfies a desired condition.

In the system according to one or more embodiments, the second inputunit may receive a condition regarding the predetermined task from theuser, and the output unit may output information that is based on thecondition. According to this aspect, the user can flexibly set acondition corresponding to a condition for a task, during the executionof work. Therefore, it is possible to adjust an output according to thedetails of work. As a result, when a task that involves a trade-offrelationship between the accuracy of work and a processing speed is tobe executed, for example, it is possible to flexibly perform adjustmentaccording to the details of work during the execution of work so that,for example, accuracy is regarded as more important, or the processingspeed is regarded as more important, without re-training the learningmodule.

In the system according to one or more embodiments, the informationoutput from the output unit may partially include information that is tobe presented to a user according to the condition. According to thisaspect, an output corresponding to the condition input by the user canbe presented to the user. Thus, it is possible to visualize the outputcorresponding to the input condition.

Also, in a system according to one aspect, the one or more externalsystems may include a camera, the input from the user received by thesecond input unit may include a condition regarding an inspectioncriterion, and the output unit may use an image of a target objectcaptured by the camera, to output an inspection result of the targetobject based on the inspection criterion. According to this aspect, wheninspecting the quality or the like of a target object using the learningmodule, it is possible to execute inspection in view of an input fromthe user.

A system according to one aspect is a system that controls operations ofa robot based on information output from the output unit. The one ormore external systems may include a sensor configured to detect acurrent orientation of the robot, the input from the user received bythe second input unit may include a condition regarding a constraint onthe operations of the robot, and the output unit may output informationfor controlling the operations of the robot in view of the currentorientation of the robot and the condition. According to this aspect,when using a learning module to control the operations of a robot, it ispossible to enable the robot to operate in view of an input from theuser.

A system according to one aspect is a system for controlling operationsof a robot based on information output from the output unit. The one ormore external systems may include a sensor configured to detect at leastone of a current position and a current orientation of the robot, theinput from the user received by the second input unit may include acondition regarding safety of the robot in avoiding an obstacle, and theoutput unit may output information for controlling the operations of therobot in view of the current position of the robot and the condition.According to this aspect, when using a learning module to control theoperations of a robot to enable the robot to avoid an obstacle, it ispossible to enable the robot to perform an avoidance action in view ofan input from the user.

In a task execution system according to one aspect, the one or moreexternal systems may include a camera, the input from the user receivedby the second input unit may include a condition regarding a part of ahuman body, and the output unit may use an image of a person captured bythe camera to determine a matching level with a specific target imagebased on the condition input by the user. According to this aspect, whenusing a learning module to search for a person captured by asurveillance camera or the like, it is possible to realize a search inview of an input from the user.

A training apparatus according to one aspect is an apparatus that trainsthe learning module included in the above-described systems, andincludes a learning control unit configured to train the learning modulebased on training data that includes first training data that isacquired from one or more external systems, and second training datathat includes data that is in the same format as a condition that isinput by the user when execution of the predetermined task is to berealized. According to this aspect, it is possible to train a learningmodule that is used by a system that uses a learning module to realizeexecution of a predetermined task.

A control method according to one aspect is a method for realizingexecution of a predetermined task, using a system that is provided witha learning module that includes a trained model that has been subjectedto predetermined training through machine learning, or a model that isequivalent to the trained model in terms of an input-outputrelationship. The method includes: a first step in which a first inputunit receives information that is acquired from one or more externalsystems, and generates at least a portion of information that is to beinput to the learning module; a second step in which the learning moduleoutputs predetermined information based on at least the informationgenerated in the first step; a third step in which an output unitacquires at least the information output in the second step, andgenerates information that is to be output from the system, theinformation output from the system being information based on whichexecution of a predetermined task is to be realized; and a fourth stepthat is performed substantially in parallel with at least one of thefirst step, the second step, and the third step, and in which an inputfrom a user is received so that information that is based on the inputfrom the user is input to at least one of the first input unit, thelearning module, and the output unit, and information that is outputfrom the output unit varies based on the input from the user. Accordingto this aspect, with a method that uses a learning module to cause asystem to execute a predetermined task, it is possible to generate anappropriate output in view of an input from a user, without re-trainingthe learning module.

A training method according to one aspect is a method for training thelearning module included in the above-described systems, including:training the learning module through machine learning based on trainingdata that includes first training data that is acquired from one or moreexternal systems, and second training data that includes data that is inthe same format as a condition that is input by the user when executionof the predetermined task is to be realized. According to this aspect,it is possible to provide a method for training a learning module thatis used by a system that uses a learning module to realize execution ofa predetermined task.

A program according to one aspect causes a computer that includes alearning module that is constituted by a trained model that has beensubjected to predetermined training through machine learning to realizeexecution of a predetermined task, or a model that is equivalent to thetrained model in terms of an input-output relationship, to execute: afirst step of receiving information that is acquired from one or moreexternal systems, and generating at least a portion of information thatis to be input to the learning module; a second step in which thelearning module outputs predetermined information based on at least theinformation generated in the first step; a third step of acquiring atleast the information output in the second step, and generatinginformation that is to be output from the computer, the informationoutput from the computer being information based on which execution of apredetermined task is to be realized; and a fourth step that isperformed substantially in parallel with at least one of the first step,the second step, and the third step, and in which an input from a useris input in at least one of the first step, the second step, and thethird step so that information that realizes execution of thepredetermined task varies based on the input from the user. According tothis aspect, with a program that uses a learning module to cause asystem to execute a predetermined task, it is possible to generate anappropriate output in view of an input from a user, without re-trainingthe learning module.

A program according to one aspect causes the computer to realize afunction of training the learning module through machine learning basedon training data that includes first training data that is acquired fromone or more external systems, and second training data that includesdata that is in the same format as a condition that is input by the userwhen execution of the predetermined task is to be realized. According tothis aspect, it is possible to provide a program for training a learningmodule that is used by a system that uses a learning module to realizeexecution of a predetermined task.

In the present specification and so on, “unit” does not simply means aphysical means or part, and may refer to a case in which the functionsof the means or part are realized by a hardware processor executingsoftware that is stored in a storage apparatus. The functions of one“unit” may be realized by two or more physical means, and the functionsof two or more “units” may be realized by one physical means.

According to one or more embodiments, it is possible to allow a user tomake adjustments according to the conditions of work, during theexecution of work, in cases where a system realizes execution of apredetermined task using a learning module that includes a trained modelor a model that is equivalent to the trained model.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a concept of an overall system thatincludes a task execution system according to one or more embodiments.

FIG. 2 is a block diagram illustrating an example of a functionalconfiguration a task execution system according to one or moreembodiments.

FIG. 3 is a diagram illustrating an example of a hardware configurationof a task execution system according to one or more embodiments.

FIG. 4 is a diagram illustrating an example of a flow of processing thatis performed by a task execution system according to one or moreembodiments.

FIG. 5 is a block diagram illustrating an example of a functionalconfiguration in a case where a task execution system is applied to animage inspection apparatus.

FIG. 6 is a diagram illustrating an example of training data.

FIG. 7 is a block diagram illustrating an example of a functionalconfiguration in a case where a task execution system is applied to agripping system.

FIG. 8 is a diagram illustrating examples of operation candidates,gripping success rates, and constraint satisfaction levels.

FIG. 9 is a diagram illustrating examples of operation determinationrules.

FIG. 10 is a diagram illustrating another embodiment of a grippingsystem.

FIGS. 11A and 11B are diagrams illustrating examples of gripping successrates and constraint satisfaction levels.

FIG. 12 is a diagram illustrating another embodiment of a grippingsystem.

FIG. 13 is a diagram illustrating an example of specification ofconditions.

FIG. 14 is a block diagram illustrating an example of a functionalconfiguration in a case where a task execution system is applied to anobstacle avoidance system of a multi-jointed robot.

FIG. 15 is a diagram illustrating examples of original target pathcandidates, avoidance success rates, and target deviation rates.

FIG. 16 is a block diagram illustrating an example of a functionalconfiguration in a case where a task execution system is applied to anobstacle avoidance system of a multi-agent system.

FIG. 17 is a schematic diagram illustrating a multi-agent system.

FIG. 18 is a block diagram illustrating an example of a functionalconfiguration in a case where a task execution system is applied to aperson search system.

FIG. 19 is a diagram illustrating an example of a matching level and aweight, for each body part.

FIG. 20 is a block diagram illustrating an example of a functionalconfiguration in a case where a task execution system is applied to aninverse kinematics model.

FIG. 21 is a diagram illustrating an example in a case where there are aplurality of solutions in inverse kinematics.

DETAILED DESCRIPTION

Hereinafter, one or more embodiments will be described in detail withreference to the drawings. Note that identical elements are denoted byidentical reference signs, and redundant description thereof is omitted.Also, the following one or more embodiments are examples that illustratethe present invention, and is not intended to limit the presentinvention to only the one or more embodiments. Furthermore, the presentinvention may be variously modified without departing from the spiritthereof.

FIG. 1 is a diagram showing a concept of an overall system 1 thatincludes a system 10 (hereinafter also referred to as “task executionsystem 10”) that realizes execution of a predetermined task according toone or more embodiments. The task execution system 10 is a system thatrealizes execution of a predetermined task, using a learning module 16.Examples of tasks that are executed may include, but are not limited to,the task of determining the quality of a product and outputs the resultof inspection on a display device, the task of outputting an operationalinstruction to a robot to instruct the robot to grip an object, and soon. Note that, as described below, the learning module 16 includes oneunit of a dedicated or general-purpose piece of hardware or softwarethat has the ability to learn through machine learning, or one unitcomposed of a given combination of such units. That is, the learningmodule 16 includes a software program that includes at least one of: alearning model that has the ability to learn through machine learning;and a trained model that has acquired a predetermined ability throughmachine learning.

The learning module 16 includes a computation apparatus that includes: astorage apparatus in which the software program is stored; and ahardware processor that reads out and executes the software program.“Realizing execution of a predetermined task” refers to a case where anexternal system 50 is caused to execute a predetermined task, and mayalso refer to a case where the task execution system 10 itself executesa predetermined task. The task execution system 10 is applicable notonly to a case where an actual system is caused to execute a task, butalso to a case where a simulator, which is a virtual system, is causedto execute a task. In such a case, targets that are to be controlled area virtual sensor, a virtual robot, a virtual system, and so on.

As shown in FIG. 1, the task execution system 10 can receive informationI_(P) that is input from a user (person) P, in addition to informationI_(S) that is input from an external system 20 on the input side(hereinafter also referred to as “input-side external system 20”) thatincludes a sensor and an external device, for example.

Examples of information I_(S) that is input from the input-side externalsystem 20 include, but are not limited to, an image such as a bitmapimage, a group of points (a point cloud), a force (an output value froma force sensor), a distance (an output value from a proximity sensor),and a robot orientation (an output value from an encoder). Examples ofinformation I_(P) that is input from a user include, but are not limitedto, requirements of work and constraint conditions concerning work.Requirements of work include, for example, a selection of conditionsthat are in a trade-off relationship, such as a degree of balancebetween accuracy and speed, and constraint conditions concerning workinclude, for example, prohibition settings, such as specification of anuntouchable part of a target object. Hereinafter, information I_(P),such as requirements and constraint conditions concerning a relevanttask, may simply be referred to as “condition (constraint)”. Thebehavior of the task execution system 10 according to one or moreembodiments is changed according to a condition input from a user. Thatis, a condition input by a user is information that changes the behaviorof the task execution system 10 (e.g. a determination result and anoperational instruction that are to be output) when the task executionsystem 10 executes a predetermined task. When input to a first inputunit 12 described below, a condition input by a user is used as acondition that changes information that is to be input to the learningmodule 16 via the first input unit 12. When input to the learning module16 described below, such a condition is used as a condition that changesinternal parameters of a trained model included in the learning module16. When input to an output unit 18 described below, such a condition isused as a condition that changes information that is to be output fromthe output unit 18. Here, to change information may be to deleteinformation.

Based on information I_(S) and information I_(P) thus received, the taskexecution system 10 executes a predetermined task, using the learningmodule 16, or outputs information O_(S) that is used to cause theexternal system 50 on the output side (hereinafter also referred to as“output-side external system 50”) to execute a predetermined task.Information O_(S) output from the task execution system 10 is passed tothe output-side external system 50 so that a predetermined task isexecuted. Information that is output from the task execution system 10may contain information O_(P) that is to be Presented to a User P.Examples of Information O_(S) that is output to the output-side externalsystem 50 include, but are not limited to, an instruction value for amotor, an operational instruction for a robot hand, an optimum grippingorientation, and an image inspection result. Information O_(P) that ispresented to a user P is information that is visualized according to acondition included in information IP input by the user P. Specifically,examples of information OP include, but are not limited to, a taskachievement rate and an intermediate result. Also, a user who inputsinformation I_(P) and a user to whom information O_(p) is presented maybe the same or different. Here, it is preferable that information O_(P)is information corresponding to a condition that is input by a user. Forexample, data corresponding to a condition input by a user andindicating the degree of fulfillment of the condition may be used. Also,for example, if information I_(P) input from a user is a degree ofbalance between accuracy and speed regarding robot operation control,instruction information (an instruction value) regarding robot operationcontrol, an expected accuracy (a possible operational error range) ofthe robot operating based on the instruction information (theinstruction value), and the time required to fulfill the operation maybe presented as information O_(P).

In one or more embodiments, when the task execution system 10 is torealize execution of a predetermined task, using the trained learningmodule 16, based on the information I_(S) input from the input-sideexternal system 20, the user P can specify information I_(P), such asrequirements and constraint conditions concerning the task. The taskexecution system 10 determines output in view of the information I_(P)specified by the user P, in addition to the information I_(S) input fromthe input-side external system 20. According to one or more embodiments,when requirements and constraint conditions such as the accuracy,execution speed, and failure tolerance of a task that is to be executedneed to be changed for each work site and each kind of work, the user Pcan obtain a desirable output corresponding to conditions such asrequirements and constraint conditions, by changing information I_(P)input by the user P, without re-training the learning module 16.

Also, if an unexpected operation occurs when the task execution system10 realizes execution of a predetermined task, the user P can adjust thebehavior of the task execution system 10 executing the task, byadjusting the information I_(P) to be input (such as input parameters).In addition, when a problem occurs, there are cases in which it is easyto identify the cause by using the information O_(P) presented to theuser P.

Note that the user P does not need to input information I_(P) every timeinformation I_(S) is input from the input-side external system 20.Instead of information I_(S) input by the user P, a predetermined valuecorresponding to a task that is to be executed, or an object that is thetarget of the task may be applied. If this is the case, for example,when a condition has changed or a condition is to be applied to a uniquework site, the user P may input a condition to the task execution system10 according to the circumstances. In this case, the predetermined valuemay have been determined based on training data that is used to train alearner 70 as described below. That is, training data includes data thatis in the same format as a condition that can be input by the user Pwhen execution of a task is to be realized using the learning module 16.Therefore, it is possible to set a value based on data that is in thesame format as a condition that is input by the user P when a task is tobe executed.

FIG. 2 is a block diagram showing an example of a functionalconfiguration of the task execution system 10 according to one or moreembodiments. FIG. 3 is a block diagram showing an example of a hardwareconfiguration of the task execution system 10 according to one or moreembodiments.

As shown in FIG. 2, the task execution system 10 includes the firstinput unit 12, a second input unit 14, the learning module 16, and theoutput unit 18. Also, as shown in FIG. 3, the task execution system 10includes a computation apparatus 61, a storage apparatus 62, an externalinterface (external I/F) 63, an input apparatus 64, and an outputapparatus 65 to realize the functions shown in FIG. 2.

The computation apparatus 61 includes a CPU (Central Processing Unit)611, which is a hardware processor, a RAM (Random Access Memory) 612, aROM (Read Only Memory) 613, and so on, and controls each constituentelement according to information processing that is to be performed. Thestorage apparatus 62 is an auxiliary storage apparatus such as a harddisk drive or a solid state drive, and stores, for example, parametersof a trained model that is included in the learning module shown in FIG.2, and programs or the like that are used to execute predeterminedprocessing that is performed using the trained model. The storageapparatus 62 stores information I_(S) that is input from the input-sideexternal system 20 (e.g. a sensor 30) and information I_(P) that isinput by a user. The storage apparatus 62 also stores programs that areused to realize execution of tasks. For example, in a case where thetask execution system 10 controls a gripping system that uses a robothand, the storage apparatus 62 stores a program for computing the pathof the robot hand, the initial value of a constraint input from the userregarding the orientation of the robot hand, and so on.

The external interface 63 is an interface that is used for connection tothe input-side external system 20 and the output-side external system50, and is configured as appropriate depending on the input-sideexternal system 20 and the output-side external system 50 that areconnected. The external interface 63 may be a communication interfacethat is used for connection to another computer via a network. The firstinput unit 12, the second input unit 14, and the output unit 18 shown inFIG. 2 include an external interface 63 that is hardware. In one or moreembodiments, the task execution system 10 is connected to the input-sideexternal system 20 and the output-side external system 50 via theexternal interface 63. The task execution system 10 reads out a programfor executing computation processing based on a trained model, loads theprogram onto the RAM 612, and interprets and executes the program usingthe hardware processor.

Note that the task execution system 10 may further be provided with, forexample, the input apparatus 64 for performing input such as a mouse ora keyboard, and the output apparatus 65 for performing output, such as adisplay or a speaker.

The task execution system 10 may further be provided with a driveapparatus 66 for reading a program stored in a recording medium, such asa CD drive or a DVD drive.

In FIG. 2 again, the first input unit 12 and the second input unit 14function as interfaces for inputting information to the task executionsystem 10. The first input unit 12 has the function of receivinginformation I_(S) that is input from the external system 20, not from aperson, such as a sensor 30 and an external device 40. On the otherhand, the second input unit 14 has the function of receiving informationthat is input from a person, i.e. information I_(P) that is input from auser of the task execution system 10.

The second input unit 14 passes the information I_(P) received from theuser to at least one of: the first input unit 12; the learning module16; and the output unit 18. At this time, the second input unit 14 maypass all or part of the information received from the user to any unit,without change, or pass information that has been generated or convertedbased on the information received from the user.

The second input unit 14 may also be provided with a memory that storesthe information I_(P) received from the user. With this configuration,the information stored in the memory is passed to any unit, and thus theuser P is saved from having to input information every time. In thisregard, it is preferable that the memory stores, in addition toinformation I_(P), a correspondence relationship with information I_(S)that is input to the first input unit 12. With this configuration, it ispossible to select an appropriate piece of information I_(P) accordingto the contents of pieces of information I_(S) that are acquired fromthe input-side external system 20. Note that the memory may be locatedinside the second input unit 14. That is, it is sufficient that the taskexecution system 10 is provided with a memory.

The first input unit 12 generates information that is to be input to thelearning module 16, based on the information received from theinput-side external system 20. Upon information being passed from thesecond input unit 14 to the first input unit 12, i.e. upon the firstinput unit 12 receiving information that is based on an input from auser, the first input unit 12 generates information that is to be inputto the learning module 16 in view of information that is based on theinput from the user as well. At this time, the first input unit 12 maypass all or part of the information received from the input-sideexternal system 20 and the second input unit 14 to the learning module16, without change, or pass information that is generated or convertedbased on the information received from the input-side external system 20and the second input unit 14.

The first input unit 12 may be provided with a state recognition unit121 and an information generation unit 122. The state recognition unit121 and the information generation unit 122 are realized by thecomputation apparatus 61 of the task execution system 10 executinginformation processing that is based on a state recognition program thatis stored in the storage apparatus 62.

The state recognition unit 121 recognizes, for example: the positionand/or orientation state of a target object that is observed by a sensor30; and the internal state of an external device 40, based onobservation information acquired from the sensor 30 and the externaldevice 40, and outputs the recognized states to the learning module 16as recognition results.

The information generation unit 122 generates new information orperforms data format conversion, based on information acquired from asensor 30, an external device 40, and the second input unit 14. In thepresent specification, generation and conversion of information maysimply be referred to as “generation of information”.

The state recognition unit 121 and the information generation unit 122may have a trained model. That is, state recognition that is based oninformation acquired from the input-side external system 20 andconversion of the information acquired from the input-side externalsystem 20 may be performed using trained models that have been generatedbased on predetermined machine learning. At this time, the staterecognition unit 121 and the information generation unit 122 function assub learning modules for achieving a predetermined task that is to beexecuted by the learning module 16.

The learning module 16 includes one unit of dedicated or multi-purposehardware or software that has the ability to learn through machinelearning, or one unit constituted by a given combination of such units.The learning module 16 also includes a trained model, and a copy or adistillated version of the trained model. Here, a copy of a trainedmodel is not limited to a model in which the internal structure of thetrained model is copied, but may be a model generated by performingadditionally training on a trained learning module that has been trainedor on a copy of the trained learning module. A distillated model is atrained model that is obtained through so-called distillation. Note thatdistillation includes training another learning model that has astructure that is different from the structure of the trained model suchthat the functions of the trained model are retained, to obtain anothertrained model that has been trained. Here, it is preferable that theother trained model (the distillated model) has a simpler internalstructure and is more suitable for deployment than the trained model onwhich the distillated model is based. Note that a copy and a distillatedversion of the trained model are not necessarily provided with theability to learn. The learning module 16 includes a predeterminedstructure that has the function of converting input to output accordingto parameters. One example of such a structure is a neural network.Therefore, in the following description, it is assumed that the learningmodule 16 is constituted by a neural network. However, the learningmodule 16 is not limited to a neural network.

In one or more embodiments, information may be input to the learningmodule 16 from the first input unit 12 and the second input unit 14.Thus, in the learning module 16, a predetermined computation isperformed based on information acquired from the input-side externalsystem 20 including a sensor 30 and an external device 40, for example,and information acquired from a user (a person), and the result ofcomputation is output in the form of a value or a pattern. Theinformation thus output is passed to the output unit 18.

If the learning module is constituted by a multilayer neural network,information input from the first input unit 12, i.e. informationacquired from the input-side external system 20, is input to the inputlayer of the neural network. In contrast, although information inputfrom the second input unit 14, i.e. information acquired from the user,may also be input to the input layer of the neural network, this is notessential, and may be input to a layer other than the input layer, suchas an intermediate layer or the output layer.

A trained model that has been trained through machine learning so as tobe able to execute a predetermined task, or to cause the external system50 to execute a predetermined task, may be employed as the learningmodule 16. This trained model can be acquired by the learner 70 throughmachine learning using training data that includes information acquiredfrom the input-side external system 20 including a sensor 30 and anexternal device 40, for example, and information acquired from a user (aperson). Alternatively, a model that is equivalent to the trained modelin terms of an input-output relationship, such as a copy or adistillated version of the trained model, may be employed as thelearning module 16. In the present specification, these models maysimply be referred to as trained models. Furthermore, the learningmodule 16 may have a plurality of trained models.

The output unit 18 is an interface for outputting information from thetask execution system 10. The output unit 18 generates information O_(S)and information O_(P), which are to be output from the task executionsystem 10, based information acquired from the learning module 16. Notethat the output unit 18 may generate either information O_(S) orinformation O_(P). If information has been passed from the second inputunit 14 to the output unit 18, i.e. if the output unit 18 has acquiredinformation that is based on input from a user, information O_(S) andinformation O_(P), which are to be output from the task execution system10, are generated in view of the information that is based on the inputfrom the user as well. At this time, the output unit 18 may output allor part of the information received from the learning module 16 and thesecond input unit 14, without change, or output information that hasbeen generated or converted based on the information received from thelearning module 16 and the second input unit 14.

The information output from the output unit 18, i.e. the informationO_(S) output from the task execution system 10, is input to theoutput-side external system 50, and a predetermined task is executed inthe output-side external system 50. Also, information O_(P), which ispart of the information output from the output unit 18, may beinformation that is presented to the user P. Here, if the learningmodule 16 is constituted by a multilayer neural network, one layer fromamong the intermediate and output layers of the neural networkpreferably has a node that outputs information that is to be presentedto the user P.

The learner 70 is a training apparatus that has the function ofacquiring a trained model that is to be used as the learning module 16.Although the learner 70 can generate the learning module 16 included inthe task execution system 10, the learner 70 is not directly included inthe task execution system 10. Machine learning for realizing executionof a predetermined task is performed in the learner 70, using trainingdata that includes information input from the input-side external system20 and information input from a user. Training data that is used inmachine learning includes an input variable, which is data correspondingto a constraint that is input by a user, and an output value, whichindicates desirability of the output corresponding to the value of theinput variable. For example, it is possible to adopt the technology oftraining a learning module by directly providing the learning modulewith training data that includes correct data that indicates a desirableoutput value corresponding to the input variable. Alternatively, it ispossible to adopt the technology of training a learning module byproviding the learning module with an evaluation function that indicatesdesirability of the output. For example, a function that can determinean evaluation value corresponding to a combination of an input and anoutput may be used as the evaluation function. The learning module 16can be generated based on the trained model or the parameters thereof(such as connection weights in the case of a neural network), acquiredby the learner 70.

Also, as shown in FIG. 2, the task execution system 10 is connected tothe input-side external system 20 that includes, for example, one ormore sensors 30 and one or more external devices 40, via a communicationnetwork. Note that each sensor 30 and each external device 40 mayindividually be regarded as one input-side external system 20, andalternatively, a combination of a given sensor 30 and a given externaldevice 40 may be regarded as one input-side external system 20. Anexample in the latter case is a robot. Furthermore, the task executionsystem 10 is connected to the output-side external system 50 via acommunication network. Information is input to the task execution system10 from the input-side external system 20 and a user, the task executionsystem 10 outputs information to the output-side external system 50, andthus a predetermined task is executed. Note that the task executionsystem 10 may be regarded as a sub system for executing a predeterminedtask, and the sub system and the output-side external system 50 thatuses information output from the sub system may be configured as oneintegrated system.

Examples of a sensor 30 include, but are not limited to, a physicalquantity sensor for detecting a physical quantity, a chemical quantitysensor for detecting a chemical quantity, and an information sensor fordetecting information. Examples of a sensor 30 may include any sensor.Examples of a physical quantity sensor include a camera that detectslight and outputs image data or video data, a heartbeat sensor thatdetects the heartbeat of a person and outputs heartbeat data, a bloodpressure sensor that detects the blood pressure of a person and outputsblood pressure data, a vital sensor such as a body temperature sensorthat detects the body temperature of a person and outputs bodytemperature data, and any other sensors that detect a physical quantityand output an electrical signal. Examples of a chemical sensor include agas sensor, a humidity sensor, an ion sensor, and any other sensors thatdetect a chemical quantity and output an electrical signal. Examples ofan information sensor include a sensor that detects a specific patternfrom statistical data and any other sensors that detect information.

Each external device 40 is constituted by a computer system, a robot, orany other various devices. Note that an external device 40 and a sensor30 may be integrated into one piece. For example, an industrial robot,which is an external device 40, has a plurality of motors (shafts) and aplurality of links (structures) that are driven by the motors (shafts).The motors and the links that are driven by the motors are connected oneafter the other, and thus a multi-jointed industrial robot is formed.Here, each motor may be integrated with an encoder, which is a sensor 30that detects the rotation angle of the motor. An external device 40 mayinclude an operation planning unit 42 and an operation generation unit44. The operation planning unit 42 plans the operation of a targetobject that is, for example, controlled by the external device 40, orthe operations of the external device 40 itself, and has the function ofcreating an operation path, which serves as an original target path. Theoperation generation unit 44 has the function of generating operationcandidates, and generating avoidance action candidates to avoidcolliding with an obstacle. Here, operation candidates and avoidanceaction candidates are expressed as predetermined numerical values and/ornumerical vectors that indicate directions in which the external device40 moves. Note that the operation planning unit 42 and the operationgeneration unit 44 may have a trained model. That is, operationcandidates and avoidance action candidates may be generated using atrained model that has been generated through predetermined machinelearning. Also, the operation planning unit 42 and the operationgeneration unit 44 may be provided in the task execution system 10. Thatis, the operation planning unit 42 and the operation generation unit 44are provided in the task execution system 10, generate operationcandidates and avoidance action candidates based on information I_(P)that is acquired from a sensor 30 and/or an external device 40, andinput the generated candidates to the learning module 16. At this time,the operation planning unit 42 and the operation generation unit 44function as sub learning modules for achieving a predetermined task thatis to be executed by the learning module 16.

Although FIG. 2 shows the input-side external system 20 and theoutput-side external system 50 as separate components, the input-sideexternal system 20 and the output-side external system 50 may beconfigured as one integrated component. For example, when a robot with arobot hand is controlled using the task execution system 10, the robothand can be both the input-side external system 20 and the output-sideexternal system 50. For example, in a case of a robot equipped with acamera, the camera and an encoder that is provided in a joint of therobot may be equivalent to sensors 30. Also, if a value of an encoder isoutput via a robot, the robot may be equivalent to an external device40. Furthermore, if a robot operates according to an operationalinstruction output from the task execution system 10, the robot may beequivalent to the output-side external system 50. In one or moreembodiments, sensor data output from a sensor 30 may be directly inputto the task execution system 10 from the sensor 30 itself, or indirectlyinput to the task execution system 10 from the sensor 30 via an externaldevice 40. Therefore, in the present specification, a sensor 30 and anexternal device 40 are collectively referred to as “the input-sideexternal system 20” without being distinguished from each other in somecases.

Note that the configuration of the task execution system 10 is notlimited to that shown in the figure. For example, any blocks from amongthe aforementioned blocks may be physically or logically integrated intoone piece, and each block may be physically or logically divided intotwo or more elements.

FIG. 4 is a diagram showing an example of a flow of processing that isperformed by the task execution system 10 according to one or moreembodiments. First, the first input unit 12 acquires information fromthe input-side external system 20 including a sensor 30 and an externaldevice 40 (step S31). The first input unit 12 acquires information fromthe second input unit 14 in some cases. If necessary, the first inputunit 12 converts the acquired information into data that is in a formatsuitable for processing that is performed by the learning module, forexample, and thereafter outputs the data to the learning module 16.

The learning module 16 performs computation processing using a trainedmodel, based on the information input from the first input unit 12 (stepS32). In some cases, information from the second input unit 14 is alsoinput to the learning module 16. In such cases, computation processingis performed based on the information input from the first input unit 12and the information input from the second input unit 14, and thecomputation results are passed to the output unit 18.

The output unit 18 generates information that is used to cause theoutput-side external system 50 to perform a predetermined task, based onthe information input from the learning module 16, and outputs thegenerated information to the output-side external system 50 (S33). Insome cases, information from the second input unit 14 is also input tothe output unit 18. In such cases, the output unit 18 generatesinformation that is used to cause the output-side external system 50 toperform a predetermined task, based on the information input from thelearning module 16 and the information input from the second input unit14. For example, if the output-side external system 50 is a robotapparatus, and the predetermined task is a predetermined operation thatis to be executed by the robot, the output unit 18 can acquire aplurality of operation candidates from the learning module 16, select apredetermined operation candidate from among the plurality of operationcandidates based on the information input from the second input unit 14,and output information to the output-side external system 50.

In parallel with the processing in steps S31 to S33, the second inputunit 14 acquires conditions that are required for the predetermined taskto be executed and that have been specified by the user P, such as therequirements and constraint conditions of the task (step S34). Thesecond input unit 14 passes the acquired information to at least one of;the first input unit 12; the learning module 16; and the output unit 18.It is preferable that to which one from among the first input unit 12,the learning module 16, and the output unit 18 the information is to bepassed is set according to, for example, what task is to be executed bythe output-side external system 50. However, this is not essential.

The following describes embodiments in which the task execution system10 is applied to an image inspection apparatus, a gripping system, anobstacle avoidance system, a person search system, and an inversekinematics model, respectively.

Embodiment 1: Image Inspection Apparatus

FIG. 5 is a block diagram showing an example of a functionalconfiguration in a case where the task execution system 10 is applied toan image inspection apparatus. An image inspection apparatus 100 in oneor more embodiments is an apparatus that performs, as a predeterminedtask, the task of determining the quality of a target object such as aproduct, using an image captured by a camera. Here, a system thatincludes a task execution system 110, a camera 130, and a display device150 is referred to as “the image inspection apparatus 100”. The taskexecution system 110 includes a first input unit 112, a second inputunit 114, a determination unit 116, and an output unit 118. Note thatthe task execution system 110, the first input unit 112, the secondinput unit 114, the determination unit 116, the output unit 118, thecamera 130, and the display device 150 are components that arerespectively equivalent to the task execution system 10, the first inputunit 12, the second input unit 14, the learning module 16, the outputunit 18, the sensor 30, and the output-side external system 50 shown inFIG. 2. That is, the last two digits of the reference numeral of eachcomponent in the image inspection apparatus 100 are the same as those ofthe reference numeral of the component corresponding thereto in FIG. 2.The same applies to the other embodiments.

In one or more embodiments, in the image inspection apparatus 100, animage of an inspection target object captured by the camera 130 is inputto the determination unit 116 via the first input unit 112. In addition,an inspection criterion, which is a condition input by the user P, isinput to the determination unit 116 via the second input unit 114.

The determination unit 116 is constituted by a trained model (e.g. atrained neural network). Upon the determination unit 116 receiving aninput image showing the external appearance of a product and inspectioncriteria, the determination unit 116 outputs an inspection result of theproduct in view of the inspection criteria specified by the user. Forexample, the inspection result is “good” or “bad”. The inspection resultoutput by the determination unit 116 is displayed on the display device150 via the output unit 118. In addition to the inspection result, theimage inspection apparatus 100 may also display information regardingthe inspection criteria specified by the user P, on the display device150. For example, a criterion regarding an inspection target object, acriterion regarding the environment in which the inspection is to beperformed, and a criterion regarding inspection determination may beinput as inspection criteria. As a criterion regarding an inspectiontarget object, at least one of: the material, size, color, reflectance,transparency, and so on of the target object can be input, for example.As a criterion regarding the environment in which the inspection is tobe performed, the degree of brightness in the environment can be input,for example. As a criterion regarding inspection determination, acriterion regarding the severity of determination of “good” or “bad”that is to be output can be input. Inspection criteria are not limitedto these examples, and a plurality of inspection criteria may be used incombination.

FIG. 6 is a diagram showing an example of training data that is providedwhen a trained model that constitutes the determination unit 116 is tobe acquired through machine learning. As shown in the figure, intraining data, each image is associated with pieces of correct datarespectively corresponding to the determination criteria. In the exampleshown in FIG. 6, input images (image 1, image 2, etc.) are images of aninspection target object. In this example, there are three levels ofdetermination criteria. Image 1 is an image that is to be determined asbeing “good” in terms of the inspection results regarding all of thecriteria 1 to 3. Image 2 is an image that is to be determined as being“good” in terms of the inspection results regarding the criteria 1 and2, and as being “bad” regarding the criterion 3. Image 3 is an imagethat is to be determined as being “good” in terms of the inspectionresult regarding the criterion 1, and as being “bad” regarding thecriteria 2 and 3. Images 4 and 5 are images that are to be determined asbeing “bad” in terms of the inspection results regarding all of thecriteria 1 to 3.

It is possible to acquire a trained model, which is to be used in theimage inspection apparatus 100, by performing supervised learningthrough which the learner is supplied with a plurality of pieces oftraining data in each of which an image is associated with pieces ofcorrect data respectively corresponding to the determination criteria,as shown in FIG. 6.

Upon a user selecting one determination criterion from among thecriteria 1 to 3, the trained model acquired as a result of learningusing the training data shown in FIG. 6 can output the inspection resultcorresponding to the selected determination criterion. In the exampleshown in FIG. 6, the criteria 1 to 3 are respectively a lax criterion, astandard criterion, and a strict criterion.

In one or more embodiments, three levels of criteria can be input by theuser P. However, as a matter of course, two levels, or four or morelevels of criteria may be provided. Also, it is possible to enable theuser P to specify a criterion using continuous values in a range such asthe range of −1 to 1, instead of using discrete numerical values such ascriteria 1, 2, 3, etc. Furthermore, it is also possible to enable theuser P to select a criterion from among labels (lax, standard, strict,etc.) prepared in advance.

In this way, it is possible to realize an image inspection apparatusthat allows the user P to flexibly select inspection criteria accordingto the details of the inspection when executing an inspection, byperforming machine learning using training data that includes inspectioncriteria in a desirable format, to acquire a trained model with whichdetermination results vary depending on the inspection criteria, andusing the learning module 16 that has the trained model thus acquired,and any inspection criteria that are input by the user P to the learningmodule 16.

Embodiment 2-1: Gripping System (1)

FIG. 7 is a block diagram showing an example of a functionalconfiguration in a case where the task execution system 10 is applied toa gripping system. A gripping system 200 in one or more embodiments is asystem for causing a robot to grip an object, and includes a taskexecution system 210, sensors 230 such as a camera and an encoder, and arobot 240 or 250. In FIG. 7, although different reference numerals 240and 250 are assigned to the robot, they actually refer to the samerobot. Also, at least one or all of the sensors 230 may be provided inthe robot 240.

The task execution system 210 includes a first input unit 212, a secondinput unit 214, a predictor 216, and an operation determination unit218. These components respectively correspond to the first input unit12, the second input unit 14, the learning module 16, and the outputunit 18 in FIG. 2.

In one or more embodiments, the gripping system 200 is configured suchthat an image that shows the robot's hand and a grip-target object,which has been captured by the camera 230, and the current orientationof the robot 240, which can be acquired from the output value of theencoder 230 mounted on a joint of the robot, are input to the predictor216 via the first input unit 212. The task execution system 210 is alsoconfigured such that a plurality of operation candidates that have beencreated by an operation generation unit 244 of the robot 240 areacquired, and the plurality of operation candidates thus acquired areinput to the predictor 216 via the first input unit 212. In addition, acondition input by the user P is input to the predictor 216 via thesecond input unit 214. Note that the operation generation unit 244 maybe provided in the task execution system 210, or provided separatelyfrom the robot 230 and the task execution system 210. In addition, theoperation generation unit 244 may have a trained model. That is,operation candidates may be generated using a trained model that hasbeen generated through predetermined machine learning. At this time, theoperation generation unit 244 functions as a sub learning module forachieving a predetermined task that is to be executed by the learningmodule 16.

As conditions that are input by the user P, constraint conditions suchas “an area that is desired to be gripped” (a grip recommendation area)and “an area that is not to be gripped” (a grip prohibition area) of agrip-target object may be specified as constraints on work, for example.

The predictor 216 predicts, based on: the current position and/ororientation of the robot, which are/is calculated using movementdirection vectors that indicate a plurality of operation candidatesacquired from the robot 240, an image input from the camera 230 servingas a sensor, and values input from the encoder 230 serving as a sensor;and constraint conditions input by the user P, gripping success rates ofthe robot's hand when the hand moves according to the respectivemovement direction vectors, and constraint satisfaction levelscorresponding to the constraint conditions input by the user P. Theoperation determination unit 218 calculates an evaluation value for eachoperation candidate, based on the gripping success rates and theconstraint satisfaction levels that have been output from the predictor216, and determines the next operation from among the operationcandidates, based on the evaluation values. Then, the operationdetermination unit 218 generates an operational instruction forrealizing the execution of the determined operation, and outputs theoperational instruction to the robot 250. In addition, although notshown in the figure, information that is based on the gripping successrates and the constraint satisfaction levels predicted by the predictor216 may be output to a display or the like and presented to the user P.

FIG. 8 is a diagram showing examples of a plurality of operationcandidates (movement direction vectors), which are input to thepredictor 216, and the respective gripping success rates and constraintsatisfaction levels of the operation candidates, which are output fromthe predictor 216, in one or more embodiments. In the figure, themovement direction vectors such as (0,0,0), (0,1,0), and (0,−1,0)indicate candidates of the next operation. In a movement directionvector (x,y,z), x indicates the amount of movement of the hand in theleft-right direction, y indicates the amount of movement of the hand inthe top-bottom direction, and z indicates the amount of rotation of thehand. For example, (0,0,0) indicates that the hand is not to be moved atthe next operation, and (0,1,0) indicates that the hand is to be movedupward by one unit amount.

Each gripping success rate indicates the probability of ultimate successin gripping in a case where the operation corresponding thereto isperformed next. Each constraint satisfaction level indicates whether ornot the constraint conditions specified by the user will be satisfied ifthe operation corresponding thereto is performed next. When theconstraint satisfaction level is “1”, the constraint conditions will besatisfied, but when the constraint satisfaction level is “0”, theconstraint conditions will not be satisfied. For example, in a casewhere the user specifies a grip prohibition area, if an operationcandidate results in the hand touching the grip prohibition area of thetarget object, the constraint satisfaction level of the operationcandidate is determined to be “0”.

The example in FIG. 8 shows that the predictor 216 has output grippingsuccess rate “0.4” and constraint satisfaction level “1” for theoperation candidate (0,0,0), and gripping success rate “0.7” andconstraint satisfaction level “0” for the operation candidate (0,1,0).That is, if the hand is not moved, the gripping success rate is only0.4, but the hand will not enter the prohibition area. On the otherhand, if the hand is moved upward by one unit amount, the grippingsuccess rate increases to 0.7, but the constraint satisfaction level is0 because the prohibition area will be gripped if gripping issuccessful.

Note that the trained model that constitutes the predictor 216 has beentrained through machine learning so that, upon receiving: an image thatshows the hand and the work target; the current orientation of the robotthat can be acquired from the value of the encoder at the joint of therobot; and a movement direction vector, the predictor 216 outputs thegripping success rate and the constraint satisfaction level when thehand will be moved according to the movement direction vector. Such atrained model can be acquired through machine learning using trainingdata in which, for example: an image that shows the hand and the worktarget; the current orientation of the robot that can be acquired fromthe value of the encoder at the joint of the robot; a movement directionvector that indicates an operation candidate; and the gripping successrate and the constraint satisfaction level when the hand will be movedaccording to the movement direction vector, are associated with eachother.

The operation determination unit 218 calculates evaluation values basedon the respective gripping success rates and constraint satisfactionlevels of the operation candidates, output from the predictor 216. InFIG. 8, an evaluation value is a gripping success rate multiplied by aconstraint satisfaction level. However, the technology of calculating anevaluation value is not limited in this way. The operation determinationunit 218 determines the next operation to be performed, based on therespective evaluation values of the operation candidates, according topredetermined operation determination rules.

FIG. 9 shows examples of operation determination rules according to oneor more embodiments. If the operation determination rules in FIG. 9 areapplied to the operation candidates in FIG. 8, condition 1 is notsatisfied because the evaluation value is 0.4<0.9 when the hand is notmoved (0,0,0), and condition 2 is also not satisfied because theevaluation value of the operation candidate (0,0,0.5) is 0.6>0.5. Thus,condition 3 is satisfied, and an operation “move in a direction thatmaximizes the success rate” is selected. Therefore, the movementdirection vector (0,0,0.5) is selected as the next operation. In thisway, the operation determination unit 218 outputs an operationalinstruction for rotating the hand by 90°, to the robot 250.

The description above illustrates an embodiment in which the predictor216 outputs gripping success rates and constraint satisfaction levels,the operation determination unit 218 calculates evaluation values basedon the gripping success rates and the constraint satisfaction levels,and an operation is determined based on the evaluation values. However,the predictor 216 may output evaluation values in view of constraintconditions input by the user, and the operation determination unit 218may determine the operation based on the evaluation values received fromthe predictor 216. If this is the case, the trained model to be used,which constitutes the predictor 216, may have been trained throughmachine learning so that, upon receiving: an image that shows the handand the work target; the current orientation of the robot that can beacquired from the value of the encoder at the joint of the robot; and amovement direction vector, the predictor 216 outputs an evaluation valueof when the hand will be moved according to the movement directionvector. In one or more embodiments, the user P inputs a griprecommendation area and/or a grip prohibition area as a condition.However, the user P may additionally input an evaluation value that isused to determine the operation. If this is the case, an evaluationvalue input by the user P is input to the operation determination unit(output unit) 218 via the second input unit 214. With thisconfiguration, the user P can set any criteria for determining whetheror not to perform a gripper open/close operation to grip an object. Inthis case, the gripping system 200 may output information that is basedon the set criteria and the result of determination to a display or thelike to present the information to the user P.

Embodiment 2-2: Gripping System (2)

FIG. 10 is a diagram showing another embodiment of the gripping system200. Embodiment 2-1 employs a configuration in which one predictor 216outputs gripping success rates and constraint satisfaction levels.However, as shown in FIG. 10, it is possible to divide the predictorinto one that outputs gripping success rates and one that outputsconstraint satisfaction levels.

In this embodiment, a predictor 216 a predicts, for a plurality ofoperation candidates acquired from the robot 240, based on the currentposition and/or orientation of the robot that are/is calculated based onan image input from the camera 230 and a value input from the encoder230, gripping success rates of the hand when the hand moves in therespective directions in the current state. A predictor 216 b predicts,for a plurality of operation candidates acquired from the robot 240,based on: the current position and/or orientation of the robot thatare/is calculated based on an image input from the camera 230 and avalue input from the encoder 230; and constraint conditions input by theuser, constraint satisfaction levels when the hand moves in therespective directions from the current state.

FIGS. 11A and 11B are diagrams showing examples of gripping successrates that are output from the predictor 216 a and constraintsatisfaction levels that are output from the predictor 216 b. FIG. 11Ais a diagram showing examples of a plurality of movement directionvectors (operation candidates) that are input to the predictor 216 a,and the respective gripping success rates of the operation candidatesthat are output from the predictor 216 a. FIG. 11B is a diagram showingexamples of a plurality of movement direction vectors (operationcandidates) that are input to the predictor 216 b, and the respectiveconstraint satisfaction levels of the operation candidates that areoutput from the predictor 216 b.

In FIG. 10 again, the operation determination unit 218 acquires therespective gripping success rates of the operation candidates from thepredictor 216 a, the respective constraint satisfaction level of theoperation candidates from the predictor 216 b, and combines them tocalculate the evaluation value for each operation candidate. Otherprocessing is the same as that of one or more embodiments shown in FIG.6, and therefore the description thereof is omitted.

By dividing the predictor that outputs gripping success rates andconstraint satisfaction levels into two, it is possible to separatelyperform learning for acquiring a trained model for predicting grippingsuccess rates, and learning for acquiring a trained model for predictingconstraint satisfaction levels. For example, a conventional predictormay be used as the predictor 216 a for predicting gripping successrates, and the predictor 216 b for predicting constraint satisfactionlevels that are based on constraint conditions input by the user may beacquired as a trained model through machine learning. In this way, witha configuration in which the predictor 216 included in the grippingsystem 200 is divided into a plurality of predictors, when addingvarious constraint conditions, for example, there is no need tore-create the predictor 216 from scratch, and it is only necessary toindividually perform machine learning for each constraint condition.Thus, it is possible to reduce the amount of training data that is usedto perform machine learning. Also, since it is only necessary to add atrained model that has been acquired for each constraint condition, thepredictor 216 can be flexibly configured.

As with the gripping system 200, when the task execution system 10 isapplied to a predetermined system and the system has a plurality ofpredictors 216, the plurality of predictors 216 preferably have at leastthe following two learning modules. Specifically, the two learningmodules are: a first learning module that performs informationprocessing using, as input data, information such as sensor data that isacquired from the external system 20; and a second learning module thatperforms information processing using, as input data, information thatis acquired from the external system 20 and information that has beenconverted from conditions input by the user P into data that is in aformat suitable for processing that is performed by the learning module.

In one or more embodiments, the first learning module is the learningmodule 216 a, which uses sensor data acquired from the sensor 230 asinput data, and outputs gripping success rates. The second learningmodule is the learning module 216 b, which uses, as input data, sensordata acquired from the sensor 230 and information that indicates a griprecommendation area and/or a grip prohibition area, which serve asconditions that are input from the user P, and outputs constraintsatisfaction levels. In this way, with the configuration including theplurality of learning modules, it is possible to separately form thelearning module 216 a, which is essential to the execution of the taskof gripping a target object using a robot, and the learning module 216b, which outputs information that indicates constraint satisfactionlevels in view of constraint conditions on the execution of the task.Thus, it is easier to selectively use an appropriate learning moduleaccording to the constraint conditions that are to be imposed on thetask.

Embodiment 2-3: Gripping System (3)

FIG. 12 is a diagram showing another embodiment of the gripping system200. Embodiment 2-1 illustrates a configuration in which a conditioninput by the user P is input to the predictor 216 via the second inputunit 214. However, as shown in FIG. 12, it is possible to employ aconfiguration in which a condition that has been input by the user P isinput to the first input unit 212.

In this embodiment, the first input unit 212 receives a plurality ofoperation candidates (original operation candidates) generated by theoperation generation unit 244 of the robot 240. On the other hand, thefirst input unit 212 also inputs constraint conditions input by the userP, via the second input unit 214. An information generation unit 2122 ofthe first input unit 212 determines whether or not each of the originaloperation candidates acquired from the robot 240 satisfies theconstraint conditions input by the user P, and passes operationcandidates that satisfy the constraint conditions to a predictor 216.The predictor 216 predicts the gripping success rate for each of theplurality of input operation candidates, and an operation determinationunit 318 determines the next operation based on the gripping successrates. Thus, it is possible to exclude operation candidates that do notsatisfy the constraint conditions input by the user P before inputtingthem to the predictor 216, which leads to a reduction in the computationtime required for the predictor 216.

FIG. 13 is a diagram showing an example in which the user P specifiesconstraint conditions in the gripping system 200 described inEmbodiments 2-1 to 2-3. For example, if a grip-target object has an areathat is not to be touched (a grip prohibition area) and an area that isdesired to be gripped (a grip recommendation area), the user P canspecify constraint conditions, on a 3D model of the grip-target objectdisplayed on a display screen of a computer, for example. To acquire atrained model in view of the constraint conditions specified by theuser, it is necessary to convert the constraint conditions input by theuser, into data that can be input to the learning model, i.e. datasuitable for computation that is performed by the learning model. In theexample shown in FIG. 13, the constraint conditions specified by theuser are converted into feature vectors, and thus the constraintconditions are converted into data in a format that can be input to aneural network. Specifically, a 2D or 3D shape of the grip-target objectis displayed on a predetermined display apparatus that is connected tothe second input unit 214. At this time, data that has been formed bydiscretizing the shape of the grip-target object using boxes (voxels)that each have a predetermined size is retained. Numerical vectors thatindicate “grippable” and numerical vectors that indicate “ungrippable”are associated with voxels that constitute the grip-target objectaccording to the grip recommendation areas and/or grip prohibition areasinput by the user via a predetermined input apparatus that is connectedto the second input unit 214. Thus, vectors with which grippable cellsand ungrippable cells can be distinguished from each other using thevectors, based on numerical vectors of the grip-target objectdiscretized using voxels and numerical vector data that is associatedwith the voxels and indicates “grippable” or “ungrippable”, and that arein a format that can be input to a neural network, may be generated.

Embodiment 3: Obstacle Avoidance System (Multi-Jointed Robot)

FIG. 14 is a block diagram showing an example of a functionalconfiguration in a case where the task execution system 10 is applied toan obstacle avoidance system of a multi-jointed robot. An obstacleavoidance system 300 in one or more embodiments is a system thatautonomously executes work while avoiding an obstacle in a dynamicenvironment. It is possible to use a degree of priority balance betweenthe likelihood of avoiding an obstacle and work efficiency as acondition input by the user regarding the requirements of work. That is,the task execution system 10 is configured to allow the user P tospecify, as a condition, a degree of balance between the probability ofsuccess in avoiding an obstacle, which is the likelihood of avoiding anobstacle, and the work speed, which is work efficiency. Also, it ispossible to specify, for example, a degree of importance and a degree ofpriority, regarding a plurality of indices including “the consumption ofenergy related to movement”, in addition to the likelihood of avoidingan obstacle and a work speed, as conditions that are input by the userP.

The obstacle avoidance system 300 includes a task execution system 310,sensors 330, and a robot 340 or 350. At least one or all of the sensors330 may be provided in the robot 340. In FIG. 14, although differentreference numerals 340 and 350 are assigned to the robot, they actuallyrefer to the same robot.

The task execution system 310 includes a first input unit 312, a secondinput unit 314, a predictor 316, and an operation determination unit318. These components respectively correspond to the first input unit12, the second input unit 14, the learning module 16, and the outputunit 18 in FIG. 2.

In one or more embodiments, the obstacle avoidance system 300 isconfigured such that a group of points, which is information regardingobstacles that exist around the robot and have been subjected to sensingperformed by a sensor 330, are input to the predictor 316 via the firstinput unit 312. It is preferable that information regarding obstacles isexpressed as numerical vectors that approximate the shapes of theobstacles, which have been subjected to sensing performed by the sensor330, using polygons and a group of points. Also, the task executionsystem 310 acquires an original target path, which is an operation paththat an operation planning unit 342 of the robot 340 has created withouttaking the presence of obstacles into account, and inputs the originaltarget path to the predictor 316 via the first input unit 312.Furthermore, the task execution system 310 acquires avoidance actioncandidates that have been generated by an operation generation unit 344of the robot 340 and are used by the robot 340 to avoid obstacles, andinputs the avoidance action candidates to the predictor 316 via thefirst input unit 312. In addition, a safety coefficient, which is aconstraint input by the user P, is input to the predictor 316 via thesecond input unit 314. Instead of transmitting avoidance actioncandidates for avoiding obstacles, the operation generation unit 344 maytransmit operation candidates that indicate in which direction the robot340 should move from the current orientation.

The predictor 316 predicts an avoidance success rate and a targetdeviation rate for each of the avoidance action candidates, and outputsevaluation values in view of the safety coefficient specified by theuser. The operation determination unit 318 determines an avoidanceaction based on the evaluation values, and outputs an operationalinstruction for realizing the determined avoidance action, to the robot350. The robot 350 executes an avoidance action based on the operationalinstruction received from the task execution system 310. In addition,the obstacle avoidance system 300 may display information that is basedon the safety coefficient specified by the user, on the display or thelike, to present the information to the user.

Also, the predictor 316 may predict an avoidance success rate and atarget deviation rate for each of the plurality of avoidance actioncandidates, and output them to the operation determination unit 318. Atthis time, the safety coefficient specified by the user is input fromthe second input unit 314 to the operation determination unit 318 (thedashed arrow in FIG. 14). The operation determination unit 318 maycalculate an evaluation value in view of the safety coefficientspecified by the user, for each of the plurality of avoidance actioncandidates, based on the avoidance success rates and the targetdeviation rates, and determine an avoidance action based on theevaluation values.

FIG. 15 is a diagram showing examples of original target path candidatesthat are input to the predictor 316, and the respective avoidancesuccess rates and target deviation rates of target path candidatesoutput from the predictor 316. In the figure, shafts 1, 2, . . . , and 6are numbers assigned to motors that constitute the joints of a six-shaftmulti-jointed robot. Signs v1, v2, . . . , and vn each indicate the typeof operation candidate (target path) of the robot. From among signs thateach indicate an operation, an arrow shows the direction of rotation ofa motor, and φ indicates that a motor does not move. Specifically, “↑”indicates that the motor is moved in the forward direction, “↓”indicates that the motor is moved in the reverse direction, and “φ”indicates that the motor is not to be moved. The direction of rotationof a motor may be expressed as a numerical vector (−1 to +1), and theamount of rotation may be continuously expressed in combination with thedirection of rotation. Also, a numerical vector may be used to expressthe acceleration (angular acceleration) of a motor instead of thedirection of rotation of a motor.

An avoidance success rate P indicates the probability of success inavoiding an obstacle when the robot performs an operation vn. A targetdeviation rate Q is an index that indicates, when the robot performs theoperation vn, how close the resulting path is to the normal path (thetarget path) in a case where there are no obstacles. The targetdeviation rate Q is, for example, an index that indicates “1” when thepath perfectly matches the operation path in a case where there are noobstacles, and indicates “0” when only the start points and the endpoints match and the intermediate paths do not match at all.

For example, in FIG. 15, the operation candidate v1 indicates anoperation that is performed to rotate the joint of the shaft 1 in theforward direction and not move the joints of the remaining shafts 2 to6, and FIG. 15 shows that the avoidance success rate and the targetdeviation rates when the operation candidate v1 is operated as the nextoperation are respectively 0.2 and 0.8. That is, in the current state,the predictor 316 outputs 0.2 and 0.8 as the avoidance success rate andthe target deviation rate of the operation candidate v1, respectively.

In one or more embodiments, the user P inputs a safety coefficient α.Which operation candidate vn is to be selected is determined based onthis safety coefficient. For example, when calculating an evaluationvalue K of an operation, using an evaluation formula: K=avoidancesuccess rate×α (safety coefficient)+target deviation rate×(1−α), theuser can determine which is to be regarded as more important, theavoidance success rate or the target deviation rate, by adjusting thesafety coefficient α. For example, in the example in FIG. 15, if thesafety coefficient α is set to 1, the operation candidate v2 with a highavoidance success rate will be selected, and if the safety coefficient αis set to 0, the operation candidate v1 with a high target deviationrate will be selected. In this way, in one or more embodiments, an inputby a person is additionally used, and thus a user can specify a tradeoffbetween safety and efficiency on the site.

Note that the predictor 316 is constituted by a trained model. With alearner, it is possible to acquire a trained model that outputs adesirable result, by performing training using a reward function withwhich the closer to a value input by a person the ratio between theavoidance success rate in a case where an avoidance action candidate isexecuted and the rate of deviation from the original target path (worktarget operation) is, the higher a reward that can be obtained is.

Embodiment 4: Obstacle Avoidance System (Multi-Agent System)

FIG. 16 is a block diagram showing an example of a functionalconfiguration in a case where the task execution system 10 is applied toa multi-agent obstacle avoidance system. An obstacle avoidance system400 according to one or more embodiments is a system for determiningpaths through which a plurality of moving robots (agents) in the samespace such as a factory or a warehouse can arrive at their respectivedestinations in the shortest time without colliding with each other, andincludes a task execution system 410 and a moving robot 440 or 450. Themoving robot 440 is provided with a camera 430. In FIG. 16, althoughdifferent reference numerals 440 and 450 are assigned to the movingrobot, they actually refer to the same moving robot.

The task execution system 410 includes a first input unit 412, a secondinput unit 414, a predictor 416, and an operation determination unit418. These components respectively correspond to the first input unit12, the second input unit 14, the learning module 16, and the outputunit 18 in FIG. 2. In one or more embodiments, the first input unit 412includes a state recognition unit 4121. This component corresponds tothe state recognition unit 121 in FIG. 2.

In the obstacle avoidance system 400 in one or more embodiments, animage of an area around the moving robot 440, captured by the camera430, is input to the state recognition unit 4121 of the first input unit412. The state recognition unit 4121 recognizes the state of the movingrobot 440 based on the image acquired from the camera 430, and outputs astate vector to the predictor 416. Also, the task execution system 410acquires an original target path (target vector) created by an operationplanning unit 442 of the robot 440, and inputs the original target pathto the predictor 416 via the first input unit 412. Furthermore, the taskexecution system 410 acquires avoidance action candidates that have beengenerated by an operation generation unit 444 of the robot 440 and areused by the robot 440 to avoid colliding with other moving robots, andinputs the avoidance action candidates to the predictor 416 via thefirst input unit 412.

FIG. 17 is a schematic diagram of a multi-agent system according to oneor more embodiments. In the example shown in the figure, there are fivemoving robots in total, and each moving robot has a state vector (p,v)that indicates the position and speed thereof. Also, each moving robothas a target vector {u0} that indicates the target position thereof.

In FIG. 16 again, the predictor 416 calculates, based on the targetvector {u0}, the state vector {p0,v0,p1,v1,p2,v2,p3,v3}, and theplurality of avoidance action candidates {↑,↓,←,→,φ} input from thefirst input unit 412, an evaluation value for each of the plurality ofavoidance action candidates, and outputs the evaluation values to theoperation determination unit 418.

In addition to the evaluation values, a safety coefficient input by theuser P is input to the operation determination unit 418 via the secondinput unit 414. The operation determination unit 418 determines anavoidance action based on the evaluation values and the safetycoefficient, and outputs an operational instruction to the robot 450. Inaddition, the obstacle avoidance system 400 may display information thatis based on the safety coefficient input by the user, on the display orthe like, to present the information to the user.

Each moving robot needs to determine the optimum action based on thecurrent state thereof and the current states of moving robotstherearound. It is possible to acquire such an action policy throughmachine learning.

Embodiment 5: Person Search System

FIG. 18 is a block diagram showing an example of a functionalconfiguration in a case where the task execution system 10 is applied toa person search system. A person search system 500 according to one ormore embodiments is a system that extracts a specific person that isindicated by a sample, from a surveillance image. One or moreembodiments allows the user P to effectively narrow down people byinputting a body part that the user P regards as more important.

The person search system 500 includes a task execution system 510, asurveillance camera 530 that captures a video to acquire frame images,an external system 540 that stores sample images, and a display device550 for displaying processing results. The task execution system 510includes a first input unit 512, a second input unit 514, adetermination unit 516, and an output unit 518.

In the person search system 500 in one or more embodiments, a videoframe image captured by the surveillance camera 530 is input to thedetermination unit 516 via the first input unit 512. Also, a sampleimage stored in the external system 540 is input to the determinationunit 516 via the first input unit 512.

The determination unit 516 determines whether or not a specific personhas been captured, based on the acquired video frame image and thesample image. In one or more embodiments, the determination unit 516 isconstituted by a plurality of learning modules. Each learning module hasbeen trained through machine learning so as to be able to determine amatching level by comparing predetermined body parts in the images. Inthis example, the determination unit 516 includes four comparators,which are respectively constituted by a neural network for comparingeyes, for comparing mouths, for comparing hair styles, and for comparingcontours. The four comparators determine the respective matching levelsof the body parts (eyes, mouths, hair styles, and contours) by comparinga person captured in the image input from the camera 530 with a personin the sample image, and outputs the respective matching levels of thebody parts.

The output unit 518 acquires the respective matching levels of the bodyparts from the determination unit 516. On the other hand, the outputunit 518 receives weights input by the user P regarding the body parts,respectively, from the second input unit 514, calculates an overallmatching level in view of the respective weights of the body parts, andoutputs the overall matching level to the display device 550. Also, theperson search system 500 may output information based on the weightsinput by the user P regarding the body parts, respectively, to thedisplay device 550.

FIG. 19 is a diagram showing an example of a matching level and aweight, for each body part, according to one or more embodiments. Theweight of each body part has been input by the user. The output unit 518calculates an overall matching level according to a predetermined logic,based on the respective matching levels of the body parts output fromthe determination unit, and the respective weights of the body partsinput by the user.

Embodiment 6: Inverse Kinematics Model

FIG. 20 is a block diagram showing an example of a functionalconfiguration in a case where the task execution system 10 is applied toan inverse kinematics model. FIG. 21 is a diagram showing an example ina case where there are a plurality of solutions in inverse kinematics.

An inverse kinematics control system 600 in one or more embodiments is asystem that is, upon being provided with an orientation of an endeffector such as a hand or a gripper, able to output a joint angle thatrealizes the orientation. When an orientation of an end effector isprovided, there may be a plurality of joint angles that realizes theorientation, as shown in FIG. 21. In one or more embodiments, anappropriate solution is output based on a condition specified by theuser such as the condition that the moving distance from the currentorientation is the minimum.

As shown in FIG. 20, the inverse kinematics control system 600 accordingto one or more embodiments includes a task execution system 610, sensors630, and a robot 640 or 650. At least one or all of the sensors 630 maybe provided in the robot 640. In FIG. 20, although different referencenumerals 640 and 650 are assigned to the robot, they actually refer tothe same robot. The task execution system 610 includes a first inputunit 612, a second input unit 614, a predictor 616, and an operationdetermination unit 618.

In one or more embodiments, the task execution system 610 acquires thecurrent orientation of the robot 640 from the sensors 630 such as anencoder, and inputs the current orientation to the predictor 616 via thefirst input unit 612. Also, the task execution system 610 acquires atarget orientation of the end effector from an operation planning unit(not shown) of the robot 640, and inputs the target orientation to thepredictor 616 via the first input unit 612. In addition, the taskexecution system 610 acquires a condition that has been input by theuser P, and inputs the condition to the predictor 616 via the secondinput unit 614.

The predictor 616 is constituted by a trained model, and outputs a jointangle that realizes the target orientation, based on the currentorientation of the robot 640 and the target orientation of the endeffector. If there are a plurality of solutions, the predictor 616selects an appropriate solution based on the condition input by the userP, and outputs the solution (joint angle) thus selected.

The operation determination unit 618 generates an operationalinstruction based on the joint angle received from the predictor 616,and outputs the operational instruction to the robot 650. The robot 650operates based on the operational instruction thus received, so that therobot 650 can control the orientation of the end effector according tothe condition specified by the user P. The inverse kinematics controlsystem 600 may output information that is based on a condition input bythe user P, to a display or the like to present the information to theuser P.

In one or more embodiments, it is possible to acquire the trained modelthat constitutes the predictor 616, by performing supervised learningthrough which the learner is supplied with sets of an orientation of anend effector and a joint angle corresponding thereto, as training data.Specifically, it is possible to generate a solution training data set bygenerating combinations of various joint angles and computingorientations of the end effector corresponding thereto, using forwardkinematics.

Furthermore, if there are a plurality of combinations of joint anglesthat correspond to the orientation of the end effector, a predeterminedevaluation index is set, and training data with which the solutionthereof maximizes the evaluation index is generated. Through supervisedlearning using the training data thus generated, it is possible toacquire a trained model that outputs a solution that maximizes aspecified evaluation function. The evaluation function can be expressedby a combination of a positioning accuracy and a movement cost, forexample. The positioning accuracy is a difference between the desiredorientation of the end effector and the orientation corresponding to thejoint angle, and the movement cost can be calculated as the amount ofmovement from the current orientation. In addition, a distance from asingular point may be used as the evaluation index.

Note that the present invention is not limited to the above-describedembodiments, and may be carried out in various forms within the scope ofthe spirit of the present invention. Therefore, the above-describedembodiments are merely illustrative in all aspects, and are not to beconstrued as limiting. For example, the above-described processing stepsmay be partially omitted, or modified so as to be performed in any orderor in parallel, to the extent that inconsistencies in terms of contentsof processing do not arise. Also, functional configurations and hardwareconfigurations in one or more embodiments are merely examples, and arenot limited to those shown in the figures.

Programs that execute various kinds of processing described in thepresent specification may be stored in a recording medium. For example,by installing the above-described programs to a computer, it is possibleto enable the computer to function as the task execution system 10.Here, the recording medium in which the above-described programs arestored may be a non-transitory recording medium. The non-transitoryrecording medium is not limited to a specific medium, and may be arecording medium such as a CD-ROM, for example.

At least one or all of the above-described embodiments can be describedas, but are not limited to, the following appendixes.

APPENDIX 1

A system that is provided with at least one memory and at least onehardware processor that is connected to the memory, and uses a learningmodule that includes a trained model that has been subjected topredetermined training through machine learning, or a model that isequivalent to the trained model in terms of an input-outputrelationship, to realize execution of a predetermined task,

wherein the hardware processor

uses a first input unit to receive information that is acquired from oneor more external systems, and generate at least a portion of informationthat is to be input to the learning module,

uses an output unit to acquire information that is output from thelearning module, and generate information that is to be output from thesystem, the information output from the system being information basedon which execution of a predetermined task is to be realized, and

receives an input from a user so that information that is based on theinput from the user is input to at least one of the first input unit,the learning module, and the output unit, and information that is outputfrom the output unit varies based on the input from the user.

APPENDIX 2

A method for controlling a task execution system that uses a learningmodule that includes a trained model that has been subjected topredetermined training through machine learning, or a model that isequivalent to the trained model in terms of an input-outputrelationship, to realize execution of a predetermined task, the methodcomprising:

a step in which at least one hardware processor uses a first input unitto receive information that is acquired from one or more externalsystems, and generate at least a portion of first information that is tobe input to the learning module;

a step in which the hardware processor uses the learning module tooutput second information for execution of a predetermined task, basedon at least the first information thus generated;

a step in which the hardware processor uses an output unit to acquire atleast the second information thus output, and generate third informationthat is to be output from the system; and

a step that is performed by the hardware processor substantially inparallel with at least one of the generation of the first information,the output of the second information, and the generation of the thirdinformation, to receive an input from a user, input information that isbased on the input from the user to at least one of the first inputunit, the learning module, and the output unit, and vary informationthat is to be output from the output unit based on the input from theuser.

1. A task execution system for executing a predetermined task, the taskexecution system comprising: a learning module including a trained modelthat has been subjected to predetermined training through machinelearning or a model that is equivalent to the trained model in terms ofan input-output relationship; a first input unit configured to receiveinformation that is acquired from one or more external systems, andgenerate at least a portion of information that is to be input to thelearning module; an output unit configured to acquire information thatis output from the learning module, and generate information that is tobe output from the system, the information output from the system beinginformation based on which execution of a predetermined task is to berealized; and a second input unit configured to receive an input from auser so that information that is based on the input from the user isinput to at least one of the first input unit, the learning module, andthe output unit, and information that is output from the output unitvaries based on the input from the user.
 2. The system according toclaim 1, wherein the second input unit receives a condition regardingthe predetermined task from the user, and the output unit outputsinformation that is based on the condition.
 3. The system according toclaim 2, wherein the information output from the output unit partiallyincludes information that is to be presented to a user according to thecondition.
 4. The system according to claim 1, wherein the learningmodule is constituted by a neural network.
 5. The system according toclaim 1, wherein the learning module generates the information to beoutput, based on information input from the first input unit andinformation input from the second input unit.
 6. The system according toclaim 1, wherein the one or more external systems include a camera, theinput from the user received by the second input unit includes acondition regarding an inspection criterion, and the output unit uses animage of a target object captured by the camera, to output an inspectionresult of the target object based on the inspection criterion.
 7. Thesystem according to claim 1, the system controlling operations of arobot based on information output from the output unit, wherein the oneor more external systems include a sensor configured to detect a currentorientation of the robot, the input from the user received by the secondinput unit includes a condition regarding a constraint on the operationsof the robot, and the output unit outputs information for controllingthe operations of the robot in view of the current orientation of therobot and the condition.
 8. The system according to claim 1, the systemcontrolling operations of a robot based on information output from theoutput unit, wherein the one or more external systems include a sensorconfigured to detect at least one of a current position and a currentorientation of the robot, the input from the user received by the secondinput unit includes a condition regarding safety of the robot inavoiding an obstacle, and the output unit outputs information forcontrolling the operations of the robot in view of the current positionof the robot and the condition.
 9. The system according to claim 1,wherein the one or more external systems include a camera, the inputfrom the user received by the second input unit includes a conditionregarding a part of a human body, and the output unit uses an image of aperson captured by the camera to determine a matching level with aspecific target image based on the condition input by the user.
 10. Atraining apparatus that trains the learning module that is included inthe system according to claim 1, comprising: a learning control unitconfigured to train the learning module based on training data thatincludes first training data that is acquired from one or more externalsystems, and second training data that includes data that is in the sameformat as a condition that is input by the user when execution of thepredetermined task is to be realized.
 11. A method for realizingexecution of a predetermined task, using a system that is provided witha learning module that includes a trained model that has been subjectedto predetermined training through machine learning, or a model that isequivalent to the trained model in terms of an input-outputrelationship, the method comprising: receiving information that isacquired from one or more external systems, and generating at least aportion of information that is to be input to the learning module, by afirst input unit; outputting predetermined information based on at leastthe generated information, by the learning module; acquiring at leastthe output information, and generating information that is to be outputfrom the system, the information output from the system beinginformation based on which execution of a predetermined task is to berealized, by an output unit; and receiving, substantially in parallelwith at least one of the receiving and the generating by the first inputunit, the outputting by the learning module, and the acquiring and thegenerating by the output unit, an input from a user so that informationthat is based on the input from the user is input to at least one of thefirst input unit, the learning module, and the output unit, andinformation that is output from the output unit varies based on theinput from the user.
 12. A method for training a learning module that isincluded in the system according to claim 11, comprising: training thelearning module through machine learning based on training data thatincludes first training data that is acquired from one or more externalsystems, and second training data that includes data that is in the sameformat as a condition that is input by the user when execution of thepredetermined task is to be realized.
 13. A non-transitorycomputer-readable recording medium storing a program for causing acomputer that includes a learning module that is constituted by atrained model that has been subjected to predetermined training throughmachine learning to realize execution of a predetermined task, or amodel that is equivalent to the trained model in terms of aninput-output relationship, to perform operations comprising: receivinginformation that is acquired from one or more external systems, andgenerating at least a portion of information that is to be input to thelearning module; outputting, by the learning module, predeterminedinformation based on at least the generated information; acquiring atleast the output information, and generating information that is to beoutput from the computer, the information output from the computer beinginformation based on which execution of a predetermined task is to berealized; and receiving, substantially in parallel with at least one ofthe receiving and the generating, the outputting, and the acquiring andthe generating, an input from a user in at least one of the receivingand the generating, the outputting, and the acquiring and the generatingso that information that realizes execution of the predetermined taskvaries based on the input from the user.
 14. A non-transitorycomputer-readable recording medium storing a program for training thelearning module that is included in the computer according to claim 13,comprising: causing the computer to realize a function of training thelearning module through machine learning based on training data thatincludes first training data that is acquired from one or more externalsystems, and second training data that includes data that is in the sameformat as a condition that is input by the user when execution of thepredetermined task is to be realized.